Chapter 12: Some additional advanced topics
From book Python Programming (Problem solving, Packages and Libraries) published by McGraw Hill Education (India) Private limited.
By:
Note the following:-
>>> symbol and therefore cannot be directly executed. If you want to execute them on IDLE or Jupyter, you need to manually remove the >>> symbol. Markdown and code cells (extension .ipynb) and then downloaded as html. If someone wants to "modify" or "extend' this document, you may ask for the original .ipynb file by sending me an e-mail at:- 999.anuraggupta@gmail.com12.2.1. Shared reference and in-place change. (Relating to function calls)
Example to show that strings are immutable so that its individual characters cannot be ‘changed’:
(In the IDLE output, the entire error message is not shown but replaced with words “ … rest of the error message…. ”.This is done to save space)
# ---ON IDLE---
>>> st1 = 'cat'
>>> st1[0] # Accessing the 1st item ie index 0 of the string
'c'
>>> st1[0] = 'm'# Error:- trying to reassign the 1st item ie index 0 of the string
… rest of the error message …
TypeError: 'str' object does not support item assignment
However, lists can be changed in-place as follows:
# ---ON IDLE---
>>> list1 = ['A', 'B', 'C', 'D']
>>> list2 = list1
>>> id (list1)
36392272
>>> id(list2) # id(list1) same as id(list2)
36392272
>>> list1[0] = 'New' #In-place change to list item at index 0
>>> list1
['New', 'B', 'C', 'D']
>>> list2 # In-place change to list1 also changes list2;
['New', 'B', 'C', 'D']
>>> id(list1) # Changing items of list does not change id() of the list. Change is “in place”
36392272
>>> id(list2)
36392272
12.2.2. Shared reference, equality and sameness
In Python, two objects may be identical in terms of their components but may or may not be the same object. This is especially true for objects, which are more complex types than simple integers and strings. Hence, if you create two identical strings by two different assignments, they still have the same identity as shown by the id() operator. Therefore, these two strings are in the same memory location even though they were created by two different assignment statements. For instance:-
# ---ON IDLE---
>>> str1 = 'Hello'# str1 and str2 created by two different
>>> str2 = 'Hello'# assignments but have same id() showing
>>> id(str1) # that they are at same memory location
36376192
>>> id(str2)
36376192
However, this is not the case with more complex objects, such as lists. Given below is an example, where two lists myL1 and myL2 have been created with two different assignment statements. Both lists have identical items but they exist in different places in memory. Hence, if you compare these two lists (myL1 and myL2) with an equality operator, that is ==, the comparison will yield a value True (The True value on an equality comparison that is, == comparison shows that these two lists have same values). But if you compare them using an is operator, the result is false because an is operator compares their id using the id() operator and checks to see if they are the same object. Please note that myL1 and myL2 are different objects with the same value. This is shown as follows:
# ---ON IDLE---
>>> myL1 = ['cat', 'dog', 'rat'] # myL1 and myL2 have same members but they are not
>>> myL2 = ['cat', 'dog', 'rat'] # the same object. This is shown by the id() operator
>>> id(myL1) # id(myL1) will be different from id(myL2)
35517224
>>> id(myL2)
35516664
>>> myL1 == myL2 # the equality operator answers the question isEqual?
True
>>> myL1 is myL2 # The is operator answers the question isSameObject?
False
>>> myL3 = myL1 # Creating a shared reference ie myL3-> same id() as myL1
>>> id(myL3)
35517224
12.2.3. Docstring (In function definition)
All built-in Python functions have a docstring attached to it. For instance, take the case of len() built-in function in Python. You can see what len.__doc__ gives. The result is shown as follows:
# ---ON IDLE---
>>> len.__doc__
'Return the number of items in a container.'
>>>
Note:- The docstring of a function can also be accessed using the help() inbuilt function. This will become clear from the following examples:
# ---ON IDLE---
>>> help(len)
Help on built-in function len in module builtins:
len(obj, /)
Return the number of items in a container.
>>>
You can write a function with a docstring. Then you can use the function, which squares a number, that is, f1() and get its docstring using f1.__doc__ . Note that you don’t have to use brackets in front of f1 here. You can also pass f1 as a parameter to the built-in help() function as follows:
# ---ON IDLE---
>>>def f1(x):
""" Takes 1 parameter and returns its square"""
return x*x
>>> f1(2)
4
>>> f1.__doc__
' Takes 1 parameter and returns its square'
>>> help(f1)
Help on function f1 in module __main__:
f1(x)
Takes 1 parameter and returns its square
>>>
12.3. Concepts related to python module, name attribute and virtual environment.
12.3.1. How Python interpreter searches for modules. (Relating to import statements)
When the Python interpreter comes across an import statement, it imports the module if the module is present in the search path. The list of directories in which the interpreter searches for a module is called the search path.
Search path: When Python tries to load a module, it tries to find it on the users’ machine. The Python interpreter searches the module in the following sequence:
(i) It searches for the module in the current directory. You can get the current working directory by using the getcwd() function of the os module as follows:
# ---ON IDLE---
>>>import os
>>> os.getcwd()
'C:\\Python34\\myScripts'
Some of the locations where the Python interpreter searches for a module are given in the sys.path attribute of the sys module. For instance, the sys.path on the author’s system gives the following:
# ---ON IDLE---
>>> sys.path
['C:/Python34/myScripts', 'C:\\Python34\\Lib\\idlelib', 'C:\\Windows\\system32\\python34.zip', 'C:\\Python34\\DLLs', 'C:\\Python34\\lib', 'C:\\Python34', 'C:\\Python34\\lib\\site-packages']
The interpreter loads a module only once even if it is imported multiple times. This is done to prevent the executable statements in the module from getting executed over and over again.
The following script creates a module named hello contained in a file named hello.py. It simply contains a print statement, a variable myV and a function myPrint():
# hello.py
print("hello") # Executable python statement
myVariable = "Variable of hello module"
def myPrint():#This is not an executable statement of hello module
print("Inside the myPrint() function of hell.py")
Now, you may import this hello module on IDLE and also access the variable myV and call the function myPrint() of this hello module as follows:
# ---ON IDLE---
>>>import hello
hello
>>> hello.myVariable
'Variable of hello module'
>>> hello.myPrint()
Inside the myPrint() function of hell.py
>>>
Note that reloading a module causes the executable statements in the module to be executed again. This is shown as follows:
# ---ON IDLE---
>>>import imp
>>> imp.reload(hello)
Hello
<module 'hello' from 'C:/Python34/myScripts\\hello.py'>
12.3.2. Problems that may arise in importing modules
(Read from the book)
See Page 278 of the book
12.3.3. Importing only some of the attributes:-
(The code for this section is not reproduced because it is too many small pieces)
12.3.4. Using import *
(The code for this section is not reproduced because it is too many small pieces)
12.3.5. Attributes with leading underscore and import *
See Page 282 of the book
Note that when you use import *, then all such attributes beginning with an underscore will not be imported. This is done on purpose. If a module writer wants to keep certain attributes as internal to the module and does not want to allow these modules to be imported using import *, then he should name all such attributes with a leading underscore (_).
However, please note that the import statement without asterisk, that is, (*) will import even those attributes with leading underscore. This will become clear from the following example.
Create a module and save it in a file named underScoreAttributes.py. In this module, you have two functions (Or attributes of the module) namely normalPrint() and _undScoPrint().
You can then import this module first using import * and then only import. The module is as follows:
# File underScoreAttributes.py
def normalPrint():
print('normalPrint() attribute...')
def _undScoPrint():
print('_undScoPrint() which is a private attribute')
Now you can import this module using the two different import statements, as shown:
# ---ON IDLE---
>>>from underScoreAttributes import* # does not import attributes with underscore
>>> normalPrint()
normalPrint() attribute...
>>> _undScoPrint()
Traceback (most recent call last):
File "<pyshell#32>", line 1, in<module>
_undScoPrint()
NameError: name '_undScoPrint' is not defined
>>>import underScoreAttributes # imports all attributes including with underscore
>>> underScoreAttributes._undScoPrint()
_undScoPrint() which is a private attribute
>>>
12.3.6. Using __file__ attribute to find location of an imported module:
Once you import a module, you can find/ or know its location using the __file__ attribute as follows:
# ---ON IDLE---
>>> myModule1.__file__ # Module name followed by .__file__ gives location
'C:\\Python34\\myScripts\\myModule1.py'
12.3.7. Using reload()
When a module is imported into a script, the executable statements in the module are executed only once. However, if you want the executable statements in a module to be re-executed, call reload. The syntax is as follows:
reload(module_name)
12.3.9. Running a Python script from the command line
See Page 284 of the book
So far, you have been using IDE like IDLE, Spyder and Jupyter notebook for running Python scripts. But you can also use the command line to run a Python script.
To run a Python script from the command line, the format is as follows:
# ---ON IDLE---
>>> python filename.py arg1 arg2 arg3 … argN
But the Python script (which is being executed) must know how to use these optional parameters. To literally catch these optional parameters, Python has an attribute called argv in the sys module. Therefore, sys.argv gives a list of strings representing the arguments (as separated by spaces) on the command line.
This will become clear from the following example, which prints:
import sys
print(sys.argv)
# sys.argv[0] gives name of the python script being executed
print('file name->', sys.argv[0])
# If optional parameters have been given by user, then len(sys.argv) > 0
if len(sys.argv) > 0:
num_opt = len(sys.argv) - 1
print('Number of optional parameters are->', num_opt)
print('List of optional parameters->', sys.argv[1:])
12.3.10. __name__ attribute
The __name__ attribute helps you to know whether a Python file (ending with extension .py) is being executed or imported.
The following script (Saved as my_module.py at location:- C:\Temp)shows how you use the attribute __name__ to check whether a Python file (You can call it a module also) is run directly or is imported. (Use your file path instead)
import sys
if __name__ == '__main__':
print("The module is being executed")
print("__name__ attribute is->", __name__)
else:
print("The module is being imported")
print("__name__ attribute is->", __name__)
But you may now import the above module. (To import this module, you must add its directory to the sys.path.append(path_to_module).
Consider the following script. When you run this script, then the module my_module is not being executed. Rather, it is being imported. Therefore, the output is as follows:
import sys
# my_module.py file is saved at C:\Temp.
# So add the folder to sys.path before importing from it
sys.path.append('C:\Temp')
import my_module
12.3.11. Getting the ‘dependency’ tree of a module
One easy way to know dependencies of “installed” packages is to use the pipdeptree utility.
You can install pipdeptree with a pip command as shown:
# ---ON COMMAND PROMPT/ ANACONDA PROMPT---
pip install pipdeptree
Now you can run this command on the command prompt. Once you do so, you will get a dependency tree of all your Python modules. This is shown as follows:
# ---ON COMMAND PROMPT/ ANACONDA PROMPT---
pipdeptree
12.3.12. Virtual environment
See Pages 287-289 of the book
12.4.3. Some common errors (Part 1)
See Pages 291-296 of the book
12.4.4. Some Common Errors in Python (Part 2)
See Pages 296-297 of the book
(The scripts relating to above topics are too small to be reproduced. Please refer to the book for scripts relating to these topics)
`12.5. Using Jupyter notebook in interactive mode
12.5.1. Installation of ipywidgets
ipywidgets can be installed on Jupyter notebook using pip as follows:
!pip install ipywidgets
To start using the widgets you need to import them from ipywidgets as follows:
from ipywidgets import widgets
12.5.2. Using widgets of ipywidgets
Once you import widgets, you get a number of UI (User Interface) elements. For instance, you can create a textbox using widgets.Text(). See the following code:
from ipywidgets import widgets
tBox = widgets.Text()
display(tBox)
Now each widget created has a value property, which can be either set or displayed. This will become clear from the following code:
from ipywidgets import widgets
tBox = widgets.Text()
display(tBox)
tBox.value = 'cat'
tBox.value
12.5.3. Getting details of a widget
You can get details of a particular widget on Jupyter by either using the question-mark token, that is, (?) or using as shown :-
# ---ON JUPYTER---
?widgets.Checkbox
The output is:-
Init signature: widgets.Checkbox(*args, **kwargs)
Docstring:
Displays a boolean `value` in the form of a checkbox.
Parameters
----------
value : {True,False}
value of the checkbox: True-checked, False-unchecked
description : str
description displayed next to the checkbox
indent : {True,False}
indent the control to align with other controls with a description. The style.description_width attribute controls this width for consistence with other controls.
Init docstring: Public constructor
File: c:\programdata\anaconda3\lib\site-packages\ipywidgets\widgets\widget_bool.py
Type: MetaHasTraits
12.5.4. The UI and event handler of a widget
A widget can be thought of as having two parts:-
If you see the help (Using ? on Jupyter), for the widget Button, you get the following:
# ---ON JUPYTER
?widgets.Button
The output is:-
Init signature: widgets.Button(*args, **kwargs)
Docstring:
Button widget.
This widget has an `on_click` method that allows you to listen for the
user clicking on the button. The click event itself is stateless.
Parameters
----------
description: str
description displayed next to the button
tooltip: str
tooltip caption of the toggle button
icon: str
font-awesome icon name
disabled: bool
whether user interaction is enabled
Init docstring: Public constructor
File: c:\programdata\anaconda3\lib\site-packages\ipywidgets\widgets\widget_button.py
Type: MetaHasTraits
So the widget Button has an on_click() method. You can get the details of this method as follows:
# ---ON JUPYTER---
?widgets.Button.on_click
The output is:-
Signature: widgets.Button.on_click(self, callback, remove=False)
Docstring:
Register a callback to execute when the button is clicked.
The callback will be called with one argument, the clicked button
widget instance.
Parameters
----------
remove: bool (optional)
Set to true to remove the callback from the list of callbacks.
File: c:\programdata\anaconda3\lib\site-packages\ipywidgets\widgets\widget_button.py
Type: function
So now you may write a script which can do the following:
from ipywidgets import widgets
myB = widgets.Button(description = 'You can click me')
display(myB)
def on_myB_click(x):
print("I was clicked")
myB.on_click(on_myB_click)
(iii)Using the interact function to create interactive UI
See Page 300 of the book
The interact() function can be used to auto generate UI Controls for function arguments.
This means that if you had a function which takes an argument, then interact() can be used to create, say a slider for this argument so that by changing the slider you can change the argument.
So to use the interact() function, you must define a function with an argument. Then you pass the function name as the first argument to the interact function and you also pass the variable as an argument to the interact function. This will be clear from the following example:
%matplotlib inline
from ipywidgets import interact
def fSquare(x):
print('x square')
return x * x
interact(fSquare, x = 10)
Note that you can have more than one variable and you can also change the range of the slider control. You can also write a script in which you can vary the frequency f and the amplitude a. The code is as follows:
%matplotlib inline
from ipywidgets import interact
import numpy as np
import matplotlib.pyplot as plt
def fSin(f, a):
X = np.linspace(0, 1, 100)
Y = a*np.sin(2*3.14* f * X)
plt.plot(X,Y)
plt.show()
interact(fSin, f = 100, a = 1)
Note that the type of slider generated depends upon the type of data that you give to the variables. If you give say a bool value, such as True or False, then accordingly, a checkbox will be automatically generated by the interact function. Similarly, if you give a string, that is, text surrounded by quotes, then a text box will automatically be generated by the interact function. If you provide a list of strings, then a dropdown will be created.
Consider the code as run on Jupyter:
from ipywidgets import interact
def f(fruits):
print('You chose->', fruits)
f_list = ['apple', 'banana', 'cherry', 'dates']
interact(f, fruits = f_list)
(iv) Brief introduction to traitlets and using them in widgets for events
See Page 302 of the book
Python widgets provide a method called observe to handle changes. The signature of the observe method is as follows:
# ---ON JUPYTER---
observe(handler, names, type)
This may appear a bit complicated, but you can have a look at the docstring of observe:
# ---ON JUPYTER---
print(widgets.Widget.observe.__doc__)
The output is:-
Setup a handler to be called when a trait changes.
This is used to setup dynamic notifications of trait changes.
Parameters
----------
handler : callable A callable that is called when a trait changes. Its
signature should be ``handler(change)``, where ``change`` is a dictionary. The change dictionary at least holds a 'type' key.
* ``type``: the type of notification.
Other keys may be passed depending on the value of 'type'. In the case where type is 'change', you also have the following keys:
* ``owner`` : the HasTraits instance
* ``old`` : the old value of the modified trait attribute
* ``new`` : the new value of the modified trait attribute
* ``name`` : the name of the modified trait attribute.
names : list, str, All
If names is All, the handler will apply to all traits. If a list of str, handler will apply to all names in the list. If a str, the handler will apply just to that name.
type : str, All (default: 'change')
The type of notification to filter by. If equal to All, then all notifications are passed to the observe handler.
The use of observe() method of a widget will become clear from the following code:
See Page 304 of the book
mySlider = widgets.IntSlider()
display(mySlider)
def on_change(change):
print('Owner->', change['owner'],'Old was->', change['old'],
'New is->',change['new'], 'Name->', change['name'])
mySlider.observe(handler = on_change, names='value')
12.6.2. pycodestyle
Pycodestyle (Formerly PEP8) is the linter tool to check the Python code against the style conventions of PEP8
To install it,do the following:
!pip install pycodestyle
Pycode style is a command line tool and may be used on command line as follows:
# ---ON COMMAND/ ANACONDA PROMPT---
> pycodestyle [options] input ...
The complete list of options is as follows:
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-v, --verbose print status messages, or debug with -vv
-q, --quiet report only file names, or nothing with -qq
--first show first occurrence of each error
--exclude=patterns exclude files or directories which match these comma separated patterns (default: .svn,CVS,.bzr,.hg,.git)
--filename=patterns when parsing directories, only check filenames matching these comma separated patterns (default: *.py)
--select=errors select errors and warnings (e.g. E,W6)
--ignore=errors skip errors and warnings (e.g. E4,W)
--show-source show source code for each error
--show-pep8 show text of PEP 8 for each error (implies --first)
--statistics count errors and warnings
--count print total number of errors and warnings to standard error and set exit code to 1 if total is not null
--max-line-length=n set maximum allowed line length (default: 79)
--max-doc-length=n set maximum allowed doc line length and perform these checks (unchecked if not set)
--hang-closing hang closing bracket instead of matching indentation of opening bracket's line
--format=format set the error format [default|pylint|<custom>]
--diff report only lines changed according to the unified diff received on STDIN
12.6.3. Using pycodestyle
Pycodestyle (Formerly PEP8) is the official linter tool to check the Python code against the style conventions of PEP8 Python. This package used to be called pep8 but was renamed to pycodestyle to reduce confusion.
The following example shows how to use pycodestyle. One of the Python library files io.py has been taken as an example. Suppose this file on your computer resides at:- C:\ProgramData\Anaconda3\Lib\io.py.
You can use the –show-source option to show the line number where the Error/ Warning has been detected. On Jupyter, the commands are as follows:
!pycodestyle --show-source C:\ProgramData\Anaconda3\Lib\io.py
12.6.4. Using pylint
Pylint is used to check for errors in the Python script. While pycodestyle only checks for code writing style, pylint checks not only for style but also errors. Pylint checks for both bugs as well as quality level.
You can try out pylint on a Python module tweepy. (This module is a popular module for working with tweets from twitter). Suppose the path to the module on the computer is:
C:\ProgramData\Anaconda3\Lib\site-packages\tweepy
Remember, on Jupyter you can run a shell command using exclamation, that is, (!)
So you can run pylint on the tweepy module on Jupyter as follows:
# ---ON JUPYTER---
# Path to tweepy module is:-
# C:\ProgramData\Anaconda3\Lib\site-packages\tweepy
# Use your path instead
!pylint C:\ProgramData\Anaconda3\Lib\site-packages\tweepy
The output is very long and hence not shown here.
12.6.5. Pylint and static code analysis in Spyder
Read the book for this topic.
See Page 308 of the book